Lecture 7

Working With Non-Binary Outcomes: Spread

Review

Where We Are

income education county
1 $120k 16 kern
2 $4k 13 kern
3 $15k 12 kern
4 $15k 13 kern
⋮
2271 $0k 20 tulare
  • We’ve talked about estimating the mean of a population.
  • And means of subpopulations, too. And differences in those.
  • We’ve been looking the incomes of California residents with and without 4-year degrees.
    • Our population is the set of all California residents meeting some age and education criteria.
    • And we’ll think of earnings as being drawn with replacement from that population.
  • To think carefully about the relationship between earnings and the population, we’ll use random variables.

Imagining a Population

income education county
1 $51k 14 sacramento
2 $0k 16 monterey
3 $0k 13 unknown
⋮
14194 $62k 12 LA
income education county
1 $120k 16 kern
2 $4k 13 kern
3 $15k 12 kern
⋮
2271 $0k 20 tulare
  • We don’t have much information about the people in our population. Except the ones in earnings.
  • But, for the sake of visualization, I’ve made some up.
    • We’ll put visualizations of this fake population on the left.
    • Visualizations of the (real) sample will be on the right.

Unbiasedness

  • We’ve shown that using sample means as estimators of population means gives us unbiased estimators.
    • That is, estimators whose expected values are the thing we want to estimate.
    • Intuitively, that means their sampling distributions are where we want them to be.
    • Like the plot on the left. Not the one on the right.
  • This is essentially necessary to get correctly calibrated interval estimates.
  • It’s more or less sufficient too.
    • Calibration via bootstrap works for every unbiased estimator we’ll talk about this semester.
    • Things are more complicated for fancy machine learning stuff, but that’s beyond the scope.

We Can Trust This Interval Estimate

\[ \mu(1)-\mu(0) \in 41.96K \pm 5.77K \qqtext{is a 95\% confidence interval} \]

  • This is a 95% confidence interval for the difference in mean income between California residents with and without 4-year degrees.
  • In 95% of surveys, it’ll contain the different in population means. That’s the point of all of this.

The NSW version

\[ \mu(1)-\mu(0) \in 1.79K \pm 1.34K \qqtext{is a 95\% confidence interval} \]

  • This is a 95% confidence interval for the difference in mean income between California residents with and without 4-year degrees.
  • In 95% of surveys, it’ll contain the different in population means. That’s the point of all of this.

The NSW version

\[ \mu(1)-\mu(0) \in 1.79K \pm 1.34K \qqtext{is a 95\% confidence interval} \]

  • This is a 95% confidence interval for the difference in mean income between treated and untreated people in the National Supported Work experiment.
  • That’s in 1978 dollars. Accounting for inflation, it’s about \(3.5\) times that in 2022 dollars.

\[ \mu(1)-\mu(0) \in 6.28K \pm 4.68K \qqtext{is a 95\% confidence interval (in 2022 dollars)} \]

What’s The Difference?

California 4-year degree vs no 4-year degree
NSW treated vs untreated (adjusted for inflation)
  • Do these intervals tell the same story about the income differences between the groups involved?
  • In my opinion, no.
  • The California interval tells us there’s a big difference between these groups in the population.
    • It’s 36.20K to 47.73K. That’s a big difference no matter which end of the interval the population difference is on.
  • The NSW interval tells us we don’t know.
    • It’s 1.60K to 10.96K in 2022 dollars.
    • That’s a smallish difference at one end of the interval and a big one at the other.
  • What we want is to be able to anticipate this and design our studies to avoid it.
  • The key to this is understanding the spread of the distribution of our estimator. That’s what we’ll talk about today.

Probability Stuff

Review: Observations as Random Variables

  • The notation \(y_j\) refers to a number.
    • It’s the income of the \(j\)th person in our population. That’s just some person.
  • The notation \(Y_i\) refers to something else.
    • It’s the income of the \(i\)th person we call in our survey. That’s not a person.
    • That’s the result of a random process—the roll of a die.
  • To summarize this result, we talk about the probability distribution of \(Y_i\).

Notation Conventions

  • Random variables are written in uppercase: \(X\), \(Y\), \(Z\), etc. Constants are written in lowercase: \(x\), \(y\), \(z\), etc.
  • We’ll use the same letter for a random variable and the value it takes on. \(x\) is a possible value of \(X\), \(y\) of \(Y\), etc.
  • Estimators are also random variables, but instead of uppercase we use a hat: \(\hat \mu\), \(\hat \sigma\), etc.
  • We’ll use the same letter for the estimator and what it’s meant to estimate. \(\hat \mu\) is an estimate of \(\mu\).

Review: Expectations

income education county
1 $51k 14 sacramento
2 $0k 16 monterey
3 $0k 13 unknown
⋮
14194 $62k 12 LA
roll income50k income education county
1 1017 1 $120k 16 kern
2 8004 0 $4k 13 kern
3 4775 0 $15k 12 kern
⋮
2271 7117 0 $0k 20 tulare
  • Often the thing we care about is the expected value of \(Y_i\). Or some related quantity.
    • The expected value of \(Y_i\) is the probability-weighted average of the values it can take on.
    • What’s nice about this is that we can think of this in ‘uncollapsed’ terms.
  • When we sample as usual, this is just the population mean.
  • The collapsed form is, in a sense, just summing in a different order.

\[ \begin{aligned} \mathop{\mathrm{E}}[Y_i] = \sum_y P(Y_i = y) \times y = \sum_y \qty(\sum_{j: y_j = y} \frac{1}{m}) \times y = \frac{1}{m}\sum_y\sum_{j:y_j = y} y = \frac{1}{m}\sum_{j=1}^m y_j \end{aligned} \]

Rules for Expectations

The law of iterated expectations

\[ E \{ E( Y \mid X ) \} \quad \text{ for any random variables $X, Y$} \]

To calculate the mean of \(Y\), we can average within subpopulations, then across subpopulations.

Irrelevance of independent conditioning variables

\[ E( Y \mid X, X' ) = E( Y \mid X ) \quad \text{ when $X'$ is independent of $X,Y$ } \]

If \(X'\) is unrelated to \(X\) and \(Y\), holding it constant doesn’t affect the relationship between them.

Linearity of Expectations

\[ E ( a Y + b Z ) = aE(Y) + bE(Z) \quad \text{ for random variables $Y, Z$ and numbers $a,b$ } \]

We can distribute expectations through sums and multiplication by constants.

Linearity of Conditional Expectations

\[ E\{ a(X) Y + b(X) Z \mid X \} = a(X)E(Y \mid X) + b(X)E(Z \mid X) \]

When we condition on \(X\), we can distribute as if functions of \(X\) were constants.

An Important Consequence.

The expectation of a product of independent random variables is the product of their expectations. \[ \mathop{\mathrm{E}}[YZ] = \mathop{\mathrm{E}}[Y]\mathop{\mathrm{E}}[Z] \qqtext{when $Y$ and $Z$ are independent} \]

This comes up a lot in variance calculations. Let’s prove it. It’ll be good practice.

\[ \begin{aligned} \mathop{\mathrm{E}}[YZ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{YZ \mid Z}] && \text{iterated expectations} \\ &= \mathop{\mathrm{E}}\qty[ Z\mathop{\mathrm{E}}\qty{Y \mid Z} ] && \text{linearity of conditional expectations} \\ &= \mathop{\mathrm{E}}\qty[ Z \mathop{\mathrm{E}}\qty{Y} ] && \text{irrelevance of independent conditioning variables} \\ &= \mathop{\mathrm{E}}\qty{Y}\mathop{\mathrm{E}}[Z] && \text{linearity of expectations} \end{aligned} \]

Spread

  • To summarize the spread of a distribution, we often talk about variance.

  • The variance is an expectation: it’s the expected squared difference between a random variable and its expectation. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y) = \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] \]

  • To put that back on the same scale as \(Y\)—instead of the scale of \(Y^2\)—we take it’s square root. That’s the standard deviation.

\[ \text{standard deviation}(Y) = \sqrt{\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y)} \]

  • We can talk about spread outright (left) — that’s (unconditional) variance.
  • Or spread within groups (right) — that’s conditional variance.

Conditional Variance

  • The conditional variance describes the spread of one random variable within groups.
  • Like the conditional expectation of the random variable, there are two related concepts.

  1. The conditional variance function. This is a function of \(x\). \[ \sigma^2(x) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X=x) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X=x] \]

  2. The conditional variance \(\sigma^2(X) = \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X)\). This is a random variable. \[ \sigma^2(X) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X] \]

The Conditional Variance Function

\[ \sigma^2(x) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X=x) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X=x] \]

  • This is the variance of \(Y\) within the subpopulation of people with \(X=x\).
    • \(\sigma(1)\) is a number describing the spread of the green dots in our population.
    • \(\sigma(0)\) is a number describing the spread of the red dots in our population.
  • It’s the conditional expectation function of the random variable \(\textcolor{blue}{\{Y-\mathop{\mathrm{E}}(Y \mid X)\}^2}\).
  • We call it ’the conditional variance function of \(Y\) given \(X=x\).

The Conditional Variance

\[ \sigma^2(X) := \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) = \mathop{\mathrm{E}}\qty[ \textcolor{blue}{\qty{ Y-\mathop{\mathrm{E}}(Y \mid X) }^2} \mid X] \]

  • This is the variance of \(Y\) within a random subpopulation of people.
    • It’s the conditional variance function evaluated at the random variable \(X\).
    • And the conditional expectation of the random variable \(\textcolor{blue}{\{Y-\mathop{\mathrm{E}}(Y \mid X)\}^2}\).
  • We call it ‘the conditional variance of \(Y\) given \(X\)’.
  • This is the sort of thing that shows up when you use the law of iterated expectations.

The Law of Total Variance

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \mathop{\mathrm{E}}\qty{\mathop{\mathrm{\mathop{\mathrm{V}}}}( Y \mid X ) } + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{\mathop{\mathrm{E}}( Y \mid X ) } \]

  • The Law of Total Variance relates Conditional and Unconditional Variance
  • It’s like the Law of Total Expectation, but for Variance
  • It’s a useful way to decompose the variance a random variable.
  • Think about where most of the variance in \(Y\) is coming from in the two populations shown below.

An Activity

  • Working with your neighbor, think about why we call it this.
  • Try to put the three terms into words.
    • \(\mathop{\mathrm{\mathop{\mathrm{V}}}}[Y]\) is the …
    • \(\mathop{\mathrm{E}}\qty{ \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) }\) is the …
    • \(\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{\mathop{\mathrm{E}}( Y \mid X)}\) is the …
  • Do this in two ways.
    1. Using ‘Random Variables Language’,
      • e.g. ‘the expected value of the conditional variance of Y given X’
    2. Using ‘Population Language’
      • e.g. ‘the variance of income (Y) within the random subpopulation with degree status X’

Proving the Law of Total Variance

Starting Point: Interpreting Variance as Excess

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y) &= \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] && \text{ the \emph{spread} idea: the mean square of a \emph{centered version} of $Y$ } \\ &= \mathop{\mathrm{E}}\qty(Y^2) - \qty{\mathop{\mathrm{E}}\qty(Y)}^2 && \text{ the \emph{excess} idea: the average amount $Y^2$ \emph{exceeds} the square of its mean}. \end{aligned} } \]

Let’s think, in terms of the plot, about why there is excess in the square.

  • Suppose \(Y\) takes on three values — the dots —with equal probability.
    • Its expectation is the average height of the black dots.
    • And its expectation, squared, is the square of this.
  • Its square \(Y^2\) takes on the the square of those values—the squares — with equal probability.
    • Its expectation is the average height of the squares.
  • Why is the second one bigger?
    • Because squaring increases big draws of Y more than it decreases small ones.
    • Visually, the line from \(Y\) to \(Y^2\) is longer when \(Y\) is bigger than when it’s smaller.
  • We’ll mostly think about variance in terms of spread, but the ‘excess’ formula comes up in calculations.

Equivalence Proof

Claim. The spread and excess formulas for variance are equivalent. \[ \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] =\mathop{\mathrm{E}}[ Y^2] - \qty{\mathop{\mathrm{E}}(Y)}^2 \]

\[ \small{ \begin{aligned} \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y) }^2 ] &= \mathop{\mathrm{E}}\qty[ Y^2 - 2 Y \mathop{\mathrm{E}}(Y) + \qty{\mathop{\mathrm{E}}(Y)}^2 ] && \text{ FOIL } \\ &= \mathop{\mathrm{E}}[ Y^2] - \mathop{\mathrm{E}}\qty[2 Y \mathop{\mathrm{E}}(Y)] + \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}(Y)}^2] && \text{Distributing expectations (linearity)} \\ &= \mathop{\mathrm{E}}[ Y^2] - 2\mathop{\mathrm{E}}(Y)\mathop{\mathrm{E}}\qty[Y] + \qty{\mathop{\mathrm{E}}(Y)}^2\mathop{\mathrm{E}}[1] && \text{Pulling constants out of expectations (linearity)} \\ &= \mathop{\mathrm{E}}[ Y^2] - 2\qty{\mathop{\mathrm{E}}(Y)}^2 + \qty{\mathop{\mathrm{E}}(Y)}^2] && \text{Recognizing a product of something with itself as a square} \\ &= \mathop{\mathrm{E}}[ Y^2] - \cancel{2}\qty{\mathop{\mathrm{E}}(Y)}^2 + \cancel{\qty{\mathop{\mathrm{E}}(Y)}^2}] && \text{Subtracting} \end{aligned} } \]

Conditional Variance as Excess

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X) &= \mathop{\mathrm{E}}\qty[ \qty{ Y - \mathop{\mathrm{E}}(Y\mid X)}^2 \mid X ] && \text{ the \emph{spread} idea: the mean square of a conditionally centered version of $Y$ } \\ &= \mathop{\mathrm{E}}\qty(Y^2 \mid X) - \qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2 && \text{ the \emph{excess} idea: the average amount $Y^2$ exceeds the square of its cond. mean} \end{aligned} } \]

  • We can think of the Conditional Variance as excess, too.
  • It’s the same as the unconditional case, but within subpopulations.

Proof

Claim. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y \mid X)]} + \textcolor{green}{\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}(Y \mid X)]} \]

  1. Rewrite the variances, conditional and unconditional, in excess form.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X) - \qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]} + \textcolor{green}{\mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}(Y \mid X)^2} - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

  1. Distribute the outer expectations in the first term.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X)] - \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]} + \textcolor{green}{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ] - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

  1. Cancel matching terms

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{\mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}(Y^2 \mid X)] \cancel{- \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]}} + \textcolor{green}{\cancel{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ]} - \qty(\mathop{\mathrm{E}}\qty{\mathop{\mathrm{E}}\qty(Y \mid X)})^2} \]

  1. Use the law of iterated expectations to simplify the remaining terms.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[Y] = \textcolor{blue}{E[Y^2] \cancel{- \mathop{\mathrm{E}}\qty[\qty{\mathop{\mathrm{E}}\qty(Y \mid X)}^2]}} + \textcolor{green}{\cancel{\mathop{\mathrm{E}}\qty[ \qty{\mathop{\mathrm{E}}(Y \mid X)^2} ]} - (\mathop{\mathrm{E}}\qty{Y})^2} \]

Variance Calculations

The Variance of The Mean

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i Y_i - \mathop{\mathrm{E}}\qty( \frac{1}{n}\sum_i Y_i ) }^2 ] && \text{definition}\\ &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i (Y_i - \mathop{\mathrm{E}}Y_i) }^2 ] && \text{Linearity} \\ &= \mathop{\mathrm{E}}\qty[ \qty{ \frac{1}{n}\sum_i Z_i }^2 ] && \text{for} \ \ Z_i = Y_i - \mathop{\mathrm{E}}Y_i \\ &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] && \text{via FOIL} \end{aligned} \]

FOIL?

  • Remember FOIL from high school algebra?
  • When we square a sum, we get all products of terms in the sum.

\[ (a+b)^2 = \textcolor[RGB]{0,0,255}{(a+b)}\textcolor[RGB]{248,118,109}{(a+b)} = \textcolor[RGB]{0,0,255}{a}\textcolor[RGB]{248,118,109}{a} + \textcolor[RGB]{0,0,255}{a}\textcolor[RGB]{248,118,109}{b} + \textcolor[RGB]{0,0,255}{b}\textcolor[RGB]{248,118,109}{a} + \textcolor[RGB]{0,0,255}{b}\textcolor[RGB]{248,118,109}{b} = a^2 + 2ab + b^2 \]

  • Same deal for bigger sums.

\[ \qty{ \sum_i Z_i }^2 = \textcolor[RGB]{0,0,255}{\sum_i Z_i}\textcolor[RGB]{248,118,109}{\sum_j Z_j} = \sum_i \sum_j \textcolor[RGB]{0,0,255}{Z_i}\textcolor[RGB]{248,118,109}{Z_j} \]

Completing the Calculation

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] \quad \text{ for } \quad Z_i = Y_i - \mathop{\mathrm{E}}Y_i \quad && \text{via FOIL} \end{aligned} \]

Activity. Complete the calculation to get a simple expression.

Hints.

  1. Use linearity to push the expectation inside the sum.
  2. Think about what \(\mathop{\mathrm{E}}Z_i Z_j\) is when \(i=j\) and \(i\neq j\).
  3. Recall that when \(i \neq j\) they’re independent.
  4. What is \(\mathop{\mathrm{E}}Z_i^2\)?

Rules. \[ \small{ \begin{aligned} \text{1.} \quad & \mathop{\mathrm{E}}\qty[ \sum_i a_i Y_i] = \sum_i a_i \mathop{\mathrm{E}}Y_i \quad \text{ for RVs $Y_i$ and constants $a_i$.} \\ \text{2.} \quad & \mathop{\mathrm{E}}\qty[ Z_i Z_j] = \mathop{\mathrm{E}}[Z_i] \mathop{\mathrm{E}}[ Z_j] \quad \text{if $Z_i$ and $Z_j$ are independent.} \\ \end{aligned} } \]

The Variance of The Mean: Answered

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}[\bar Y] &= \mathop{\mathrm{E}}\qty[ \frac{1}{n^2}\sum_i \sum_j Z_i Z_j ] && \text{via FOIL} \\ &= \frac{1}{n^2} \sum_i \sum_j \mathop{\mathrm{E}}Z_i Z_j && \text{via Linearity } \\ &= \frac{1}{n^2} \sum_i\sum_j \begin{cases} \mathop{\mathrm{E}}Z_i^2 & \text{if } i=j \\ \mathop{\mathrm{E}}Z_i \mathop{\mathrm{E}}Z_j & \text{if } i\neq j \end{cases} && \text{using the independence of $Z_i$ and $Z_j$} \\ &= \frac{1}{n^2} \sum_i\sum_j \begin{cases} \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i) & \text{if } i=j \\ 0 \times 0 & \text{if } i\neq j \end{cases} && \text{since $Z_i=Y_i-\mathop{\mathrm{E}}Y_i$} \\ &= \frac{1}{n^2} \times n \times \mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i) = \frac{\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i)}{n} = \frac{\sigma^2}{n}. && \text{counting the copies of $\mathop{\mathrm{\mathop{\mathrm{V}}}}(Y_i)$ in the sum} \\ \end{aligned} \]

The Variance of the Subsample Mean

Claim. The variance of our point estimate is the expected value of the subpopulation variance divided by the number of people in the subsample. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1)] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{N_1} ] \quad \text{ for } \quad N_1 = \sum_{i=1}^n X_i \]

  • This is like what we saw in the unconditional case, with a twist.
  • The number of people in the subsample is random, so instead of \(1/n\), we have \(\mathop{\mathrm{E}}[ 1/N_1 ]\).
  • Let’s prove it.
    • It’s not hard, but it’s a bit long. The key is organization.
    • We’ll break things down into manageable pieces and focus on one piece at a time.

Step 1. Total Variance

  • We’ll start by using the Law of Total Variance, conditioning on \(X_1 \ldots X_n\).
  • And then we’ll have to calculate two terms.
    1. The expected value of the conditional variance
    2. The variance of the conditional expectation \[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n }] + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n} ] \\ \end{aligned} \]
  • Plugging in the definitions of variance and conditional variance, these terms are

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n} &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} ]^2 \mid X_1 \ldots X_n) \\ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}] &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} - \mathop{\mathrm{E}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}]}^2 ) \\ \end{aligned} \]

  • We see that the conditional expectation of our estimator comes up in both terms.
  • I’ve claimed it’s just \(\mu(1)\). Let’s take a look at that.

Step 2. The Estimator’s Conditional Mean

  • To calculate the conditional mean of our estimator, we’ll revisit an argument we’ve seen before.
  • We’ll look at our argument that our argument that the subsample mean \(\hat\mu(1)\) is unbiased.
    • We’ll see that it’s conditionally unbiased.
    • That is, its expected value is \(\mu(1)\), the thing we want to estimate
    • conditional on the values of the covariates \(X_1 \ldots X_n\).

\[ \begin{aligned} \mathop{\mathrm{E}}\qty[\hat\mu(1) \mid X_1 \ldots X_n] &=\mathop{\mathrm{E}}\qty[\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} \mid X_1 \ldots X_n] \\ &=\frac{\sum_{i=1}^{n} X_i \mathop{\mathrm{E}}\qty{ Y_{i} \mid X_1 \ldots X_n}}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i\mathop{\mathrm{E}}\qty{ Y_{i} \mid X_i}}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i \mu(X_i)}{\sum_{i=1}^{n}X_{i}} \\ &=\frac{\sum_{i=1}^{n}X_i\mu(1)}{\sum_{i=1}^{n}X_{i}} \\ &=\mu(1) \times \frac{\sum_{i=1}^{n}X_i}{\sum_{i=1}^{n}X_{i}} = \mu(1) \end{aligned} \]

Step 3. Simplifying Our Total Variance Decomposition

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n }] + \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n} ] \end{aligned} \]

where

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty{ \hat \mu(1) \mid X_1 \ldots X_n} &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} ]^2 \mid X_1 \ldots X_n) \\ &= \mathop{\mathrm{E}}\qty( \qty[ \hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n) \\ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}] &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}} - \mathop{\mathrm{E}}\qty[\mathop{\mathrm{E}}\qty{\hat \mu(1) \mid X_1 \ldots X_n}]}^2 ) \\ &= \mathop{\mathrm{E}}\qty( \qty{\textcolor{blue}{\mu(1)} - \mathop{\mathrm{E}}\qty[\mu(1)] }^2 ) = 0\\ \end{aligned} \]

so

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty[\hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n }] \]

Step 4. Rewriting as a Sum of Mean-Zero Random Variables

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty[\hat \mu(1) - \textcolor{blue}{\mu(1)} ]^2 \mid X_1 \ldots X_n }] \]

  • Now we’ll substitute in definitions and simplify.
  • We’ll end having to work with the conditional mean of a squared sum.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\hat \mu(1)-\mu(1))^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} - \mu(1))^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i Y_{i}}{\sum_{i=1}^{n}X_{i}} - \frac{\mu(1)\sum_{i=1}^n X_i}{\sum_{i=1}^n X_i })^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{E}}\qty{ \qty(\frac{\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)}}{\sum_{i=1}^{n}X_{i}})^2 \mid X_1 \ldots X_n }] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)})^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \end{aligned} } \]

Step 5

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] = \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}X_i \qty{Y_{i}-\mu(1)})^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \]

  • To work with this squared sum, we do our FOIL-like expansion. This is …

\[ \small{ \begin{aligned} \qty(\sum_{i}Z_i)^2 = \qty(\sum_{i}Z_i) \qty(\sum_{j}Z_i) = \sum_i\sum_j Z_i Z_j \quad && \text{ for } \quad Z_i = X_i \qty{Y_{i}-\mu(1)} \\ \text{or equivalently, using the \textbf{indicator trick}} && \text{ for } \quad Z_i = X_i\qty{Y_i - \mu(X_i)} \end{aligned} } \]

  • This lets us push the conditional mean all the way into the sum, so it meets the individual terms.
  • And when we’ve done that, we can use the irrelevance of independent conditioning variables to simplify.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \qty(\sum_{i=1}^{n}Z_i )^2 \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\mathop{\mathrm{E}}\qty{ \sum_{i=1}^{n}\sum_{j=1}^n Z_i Z_j \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_1 \ldots X_n}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \\ \end{aligned} } \]

Step 6

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \quad \text{ for } \quad Z_i=Y_i-\mu(X_i) \end{aligned} \]

  • Let’s take a look at one of the terms in the numerator, \(\mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}\).
    • We’re going to show that the cross-terms — the ones for \(i \neq j\) — are zero.
    • This will let us drop them from the sum.
    • We’ll use the fact that \(Z_i\) and \(Z_j\) are independent for \(i \neq j\).
  • Let’s rewrite a cross-term using our usual stuff
    • the law of iterated expectations, conditioning on \(Z_i\)
    • linearity of expectations;
    • irrelevance of independent conditioning variables.

\[\small{ \begin{aligned} \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j} &= \mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}(Z_i Z_j \mid Z_i,X_i,X_j) } \\ &= \mathop{\mathrm{E}}\qty{ Z_i \mathop{\mathrm{E}}(Z_j \mid X_i,X_j) } \\ &= \mathop{\mathrm{E}}\qty{ Z_i \textcolor{blue}{\mathop{\mathrm{E}}(Z_j \mid X_j)} } \quad \text{ where } \\ \textcolor{blue}{\mathop{\mathrm{E}}(Z_j \mid X_j)} &= \mathop{\mathrm{E}}\qty[X_j \qty{Y_j - \mu(X_j)}] \\ &= \mathop{\mathrm{E}}\qty{ \mathop{\mathrm{E}}\qty[X_j \qty{Y_j - \mu(X_j)} \mid X_j] } \\ &= \mathop{\mathrm{E}}\qty[X_j \qty{ \mathop{\mathrm{E}}\qty[Y_j \mid X_j] - \mu(X_j)}] \\ &= \mathop{\mathrm{E}}\qty[X_j \cross{\qty{ \mu(X_j) - \mu(X_j)}}] = 0. \\ \end{aligned} } \]

Step 7

\[ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n}\sum_{j=1}^n \mathop{\mathrm{E}}\qty{Z_i Z_j \mid X_i, X_j}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \quad \text{ for } \quad Z_i=Y_i-\mu(X_i) \\ &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} \mathop{\mathrm{E}}\qty{Z_i^2 \mid X_i}}{\qty(\sum_{i=1}^{n}X_{i})^2}] \quad \text{because cross-terms are zero}. \end{aligned} \]

  • All we’ve got to do now is recognize what we have.
  • Let’s look at a term in the numerator.
    • We’ll use iterated expecations and linearity.
    • Then the indicator trick.

\[ \small{ \begin{aligned} \mathop{\mathrm{E}}\qty[Z_i^2 \mid X_i] &= \mathop{\mathrm{E}}\qty[ X_i\qty{Y_i - \mu(X_i)}^2 \mid X_i] = X_i\mathop{\mathrm{E}}\qty[\qty{Y_i - \mu(X_i)}^2 \mid X_i] = X_i\mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[Y_i \mid X_i] = X_i \sigma^2(X_i) = X_i\sigma^2(1) \end{aligned} } \]

  • We finish up by substituting this back into what we started with.

\[ \small{ \begin{aligned} \mathop{\mathrm{\mathop{\mathrm{V}}}}\qty[ \hat \mu(1) ] &= \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} \mathop{\mathrm{E}}\qty{Z_i^2 \mid X_i}}{\qty(\sum_{i=1}^{n}X_{i})^2}] = \mathop{\mathrm{E}}\qty[ \frac{\sum_{i=1}^{n} X_i \sigma^2(1)}{\qty(\sum_{i=1}^{n}X_{i})^2}] = \mathop{\mathrm{E}}\qty[ \frac{\cancel{\sum_{i=1}^{n} X_i }\sigma^2(1)}{\qty(\sum_{i=1}^{n}X_{i})^{\cancel{2}}}] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{\sum_{i=1}^{n}X_{i}}] = \sigma^2(1) \times \mathop{\mathrm{E}}\qty[ \frac{1}{N_1}]. \end{aligned} } \]

The Variance of Differences in Means

Claim. The variance of the difference in subsample means is the sum of the variances of the subsample means themselves. \[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0)] = \mathop{\mathrm{E}}\qty[ \frac{\sigma^2(1)}{N_1} + \frac{\sigma^2(0)}{N_0}] \quad \text{ for } \quad N_x = \sum_{i=1}^n 1_{x}(X_i) = \begin{cases} 1 &\text{if} X_i=x \\ 0 &\text{otherwise}\end{cases} \]

Prove it. This is about understanding the logic of the proof we just did. Start with the law of total variance.

\[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0)] = \mathop{\mathrm{E}}\qty[ \mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n] ] + \mathop{\mathrm{\mathop{\mathrm{V}}}}[\mathop{\mathrm{E}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n] ] \]

Here are some things to think about.

  • What is the conditional expectation of the estimator given \(X_1 \ldots X_n\), i.e. \(\mathop{\mathrm{E}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n]\)? And what is its variance?
  • \(\mathop{\mathrm{\mathop{\mathrm{V}}}}[\hat\mu(1) - \hat\mu(0) \mid X_1 \ldots X_n]\) is the conditional mean \(\mathop{\mathrm{E}}[Z^2 \mid X_i]\) of some random variable \(Z\).
  • What is that random variable?
  • Can we write it in a way so that, when we square it, the cross terms are zero?
  • Hint: \(X_i(1-X_i)=0\).